Head pose mixing of audio files

ABSTRACT

Examples of wearable devices that can present to a user of the display device an audible or visual representation of an audio file comprising a plurality of stem tracks that represent different audio content of the audio file are described. Systems and methods are described that determine the pose of the user; generate, based on the pose of the user, an audio mix of at least one of the plurality of stem tracks of the audio file; generate, based on the pose of the user and the audio mix, a visualization of the audio mix; communicate an audio signal representative of the audio mix to the speaker; and communicate a visual signal representative of the visualization of the audio mix to the display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/318,782, filed on May 12, 2021, entitled “HEAD POSE MIXING OF AUDIOFILES,” which is a continuation of U.S. application Ser. No. 16/861,057,filed on Apr. 28, 2020, entitled “HEAD POSE MIXING OF AUDIO FILES,”which is a continuation of U.S. application Ser. No. 15/264,164, filedon Sep. 13, 2016, entitled “HEAD POSE MIXING OF AUDIO FILES,” whichclaims the benefit of priority under 35 U.S.C. § 119(e) to U.S.Provisional Application No. 62/219,604, filed on Sep. 16, 2015, entitled“HEAD POSE MIXING OF AUDIO FILES,” and to U.S. Provisional ApplicationNo. 62/289,085, filed on Jan. 29, 2016, entitled “HEAD POSE MIXING OFAUDIO FILES.” All aforementioned applications are hereby incorporated byreference herein in their entireties.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and in particular to systems formixing audio files based on a pose of a user.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” “augmentedreality” or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; an mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Examples of a wearable device that can present to a user of the displaydevice an audible or visual representation of an audio file aredescribed. The audio file comprises a plurality of stem tracks thatrepresent different audio content of the audio file.

An embodiment of a wearable device comprises non-transitory memoryconfigured to store an audio file comprising a plurality of stem tracks,with each stem track representing different audio content of the audiofile; a sensor configured to measure information associated with a poseof the user of the wearable device; a display configured to presentimages to an eye of the user of the wearable device; a speakerconfigured to present sounds to the user of the wearable device; and aprocessor in communication with the non-transitory memory, the sensor,the speaker, and the display. The processor is programmed withexecutable instructions to: determine the pose of the user; generate,based at least partly on the pose of the user, an audio mix of at leastone of the plurality of stem tracks of the audio file; generate, basedat least partly on the pose of the user and the audio mix, avisualization of the audio mix; communicate an audio signalrepresentative of the audio mix to the speaker; and communicate a visualsignal representative of the visualization of the audio mix to thedisplay.

In another aspect, a method for interacting with an augmented realityobject is described. The method is performed under control of a hardwarecomputer processor. The method comprises generating an augmented realityobject for interaction by a user of the wearable system; detectinggestures of a user while the user interacts with the interface;associating the detected gestures with a modification to acharacteristic of the augmented reality object; and modifying theaugmented reality object in accordance with the modification to thecharacteristic of the augmented reality object. A wearable system caninclude a processor that performs the method for interacting with theaugmented reality object.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIG. 4 schematically illustrates an example of a waveguide stack foroutputting image information to a user.

FIG. 5 shows example exit beams that may be outputted by a waveguide.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield.

FIG. 7 is a block diagram of an example of a wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 is a block diagram of another example of a wearable system.

FIG. 10 is a process flow diagram of an example of a method fordetermining user input to a wearable system.

FIG. 11 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIGS. 12-14 schematically illustrate examples of user interfaces whichpresent to a user of a wearable system visualizations of multiple steamtracks of an audio file, where the audio file is dynamically mixed basedat least in part on the user's pose.

FIG. 15 illustrates an example of a 3D user interface which showsdifferent visual graphics at different depths in the user's environment.

FIGS. 16A and 16B illustrate examples of directionalities of soundsources

FIG. 17 illustrates an example of creating a sound collage effect.

FIG. 18 illustrates an example process of presenting an audio filevisually and audibly.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION Overview

Audio files can include multiple stem tracks that represent an audiosignal for, e.g., voice, drum, guitar, bass, or other sounds. A stemtrack may be associated with multiple instruments such as a group ofdrums or a quartet of instruments, or be associated with a single sourceof sound such as voice or one musical instrument. A single stem trackcan represent a mono, stereo, or surround sound track. The audio filecan include 1, 2, 3, 4, 5, 6, 8, 10, 12 or more stem tracks. In additionto the stem tracks, the audio file can also include a master track forstandard playback.

A user may want to interact with stem tracks in an audio file andgenerate new audio files by mixing the stem tracks. However, existinguser interfaces are often cumbersome for this task because theytypically do not provide visualizations to the stem tracks and oftenrequire professional skills to combine multiple stem tracks.

The wearable system described herein is directed to solving this problemby providing visual graphics associated with stem tracks. For example, avisual graphic associated with a stem track may be a graphicalrepresentation of the musical instrument used for that stem track. Thevisual graphic may also be a virtual human if the stem track isassociated with a voice.

The wearable system can allow users to easily interact with the stemtracks using poses (such as head pose, body pose, eye pose, or handgestures). For example, a user can mix multiple stem tracks in the audiofile or mix the stem tracks across multiple audio files by moving hishands or changing his head's position. The user can also modify an audiofile, for example, by adjusting a stem track (such as adjusting thevolume of the stem track) or by replacing a stem track with another stemtrack. In some embodiments, a certain mix of the stem tracks may beassociated with a location in the user's environment. As the user movesto a location in the environment, the wearable system may play the sound(or a mixture of sounds) associated with that location. Additionalexamples of interacting with the stem tracks are further described withreference to FIGS. 12-18 .

Although the examples herein are described with reference to audiofiles, the wearable system can also be configured to allow similar userinteractions with video files, or a combination of audio and video files(such as where a video file comprises an audio sound track).

3D Display

The wearable system can be configured to present a three-dimensional(3D) user interface for a user to interact with virtual content such asvisualization of stem tracks in an audio file. For example, the wearablesystem may be part of a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it isdesirable for each point in the display's visual field to generate theaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane and/or based on observingdifferent image features on different depth planes being out of focus.As discussed elsewhere herein, such depth cues provide credibleperceptions of depth.

FIG. 2 illustrates an example of wearable system 200. The wearablesystem 200 includes a display 220, and various mechanical and electronicmodules and systems to support the functioning of display 220. Thedisplay 220 may be coupled to a frame 230, which is wearable by a user,wearer, or viewer 210. The display 220 can be positioned in front of theeyes of the user 210. The display 220 can comprise a head mounteddisplay (HMID) that is worn on the head of the user. In someembodiments, a speaker 240 is coupled to the frame 230 and positionedadjacent the ear canal of the user (in some embodiments, anotherspeaker, not shown, is positioned adjacent the other ear canal of theuser to provide for stereo/shapeable sound control). As furtherdescribed with reference to FIGS. 12-16 , the wearable system 200 canplay an audio file to the user via the speaker 240 and present 3Dvisualizations of various stem tracks in the sound file using thedisplay 220.

The wearable system 200 can also include an outward-facing imagingsystem 464 (shown in FIG. 4 ) which observes the world in theenvironment around the user. The wearable system 100 can also include aninward-facing imaging system 462 (shown in FIG. 4 ) which can track theeye movements of the user. The inward-facing imaging system may trackeither one eye's movements or both eyes' movements. The inward-facingimaging system may be attached to the frame 230 and may be in electricalcommunication with the processing modules 260 and/or 270, which mayprocess image information acquired by the inward-facing imaging systemto determine, e.g., the pupil diameters and/or orientations of the eyesor eye pose of the user 210.

As an example, the wearable system 200 can use the outward-facingimaging system 464 and/or the inward-facing imaging system 462 toacquire images of a pose of the user. The images may be still images,frames of a video, or a video, in combination or the like. The pose maybe used to mix stem tracks of an audio file or to determine which audiocontent should be presented to the user.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system and/or theoutward-facing imaging system), microphones, inertial measurement units(IMUs), accelerometers, compasses, global positioning system (GPS)units, radio devices, and/or gyroscopes; and/or b) acquired and/orprocessed using remote processing module 270 and/or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 and/or 264, such as viawired or wireless communication links, to the remote processing module270 and/or remote data repository 280 such that these remote modules areavailable as resources to the local processing and data module 260. Inaddition, remote processing module 280 and remote data repository 280may be operatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data and/or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

For example, the remote data repository 280 can be configured to storecontent of an audio file such as information associated with the stemtracks. The local processing and data module 260 and/or the remoteprocessing module 270 can detect a user's pose, such as the user'sdirection of gaze. The processing modules 260 and 270 can communicatewith the remote data repository 280 to obtain the stem tracks andgenerate visualizations of the stem tracks in the user's direction ofgaze. The processing modules 260 and 270 can further communicate withthe display 220 and present the visualizations to the user.

The human visual system is complicated and providing a realisticperception of depth is challenging. Without being limited by theory, itis believed that viewers of an object may perceive the object as beingthree-dimensional due to a combination of vergence and accommodation.Vergence movements (i.e., rolling movements of the pupils toward or awayfrom each other to converge the lines of sight of the eyes to fixateupon an object) of the two eyes relative to each other are closelyassociated with focusing (or “accommodation”) of the lenses of the eyes.Under normal conditions, changing the focus of the lenses of the eyes,or accommodating the eyes, to change focus from one object to anotherobject at a different distance will automatically cause a matchingchange in vergence to the same distance, under a relationship known asthe “accommodation-vergence reflex.” Likewise, a change in vergence willtrigger a matching change in accommodation, under normal conditions.Display systems that provide a better match between accommodation andvergence may form more realistic and comfortable simulations ofthree-dimensional imagery.

FIG. 3 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes. With reference toFIG. 3 , objects at various distances from eyes 302 and 304 on thez-axis are accommodated by the eyes 302 and 304 so that those objectsare in focus. The eyes 302 and 304 assume particular accommodated statesto bring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 306, with has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 302 and 304, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 302 and 304 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for ease of illustration, it will be appreciated that thecontours of a depth plane may be curved in physical space, such that allfeatures in a depth plane are in focus with the eye in a particularaccommodated state. Without being limited by theory, it is believed thatthe human eye typically can interpret a finite number of depth planes toprovide depth perception. Consequently, a highly believable simulationof perceived depth may be achieved by providing, to the eye, differentpresentations of an image corresponding to each of these limited numberof depth planes.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers and/orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b and/or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 and/or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective and/or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top and/or bottom major surfaces,and/or may be disposed directly in the volume of the waveguides 440 b,438 b, 436 b, 434 b, 432 b. In some embodiments, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in alayer of material that is attached to a transparent substrate to formthe waveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some otherembodiments, the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be amonolithic piece of material and the light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a may be formed on a surface and/or inthe interior of that piece of material.

With continued reference to FIG. 4 , as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration provides as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some alternative embodiments, either or both may bedynamic using electro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE's have a relatively low diffractionefficiency so that only a portion of the light of the beam is deflectedaway toward the eye 410 with each intersection of the DOE, while therest continues to move through a waveguide via total internalreflection. The light carrying the image information is thus dividedinto a number of related exit beams that exit the waveguide at amultiplicity of locations and the result is a fairly uniform pattern ofexit emission toward the eye 304 for this particular collimated beambouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets can be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet can be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, the number and distribution of depth planes and/ordepth of field may be varied dynamically based on the pupil sizes and/ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane not discernible because the location of that plane isbeyond the depth of focus of the eye may become discernible and appearmore in focus with reduction of pupil size and commensurate increase indepth of field. Likewise, the number of spaced apart depth planes usedto present different images to the viewer may be decreased withdecreased pupil size. For example, a viewer may not be able to clearlyperceive the details of both a first depth plane and a second depthplane at one pupil size without adjusting the accommodation of the eyeaway from one depth plane and to the other depth plane. These two depthplanes may, however, be sufficiently in focus at the same time to theuser at another pupil size without changing accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size and/or orientation, or upon receiving electrical signalsindicative of particular pupil sizes and/or orientations. For example,if the user's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 may beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between onand off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)and the imaging system 464 is sometimes referred to as an FOV camera.The entire region available for viewing or imaging by a viewer may bereferred to as the field of regard (FOR). The FOR may include 471steradians of solid angle surrounding the wearable system 400. In someimplementations of the wearable system 400, the FOR may includesubstantially all of the solid angle around a user of the display system400, because the user can move their head and eyes to look at objectssurrounding the user (in front, in back, above, below, or on the sidesof the user). Images obtained from the outward-facing imaging system 464can be used to track gestures made by the user (e.g., hand or fingergestures), detect objects in the world 470 in front of the user, and soforth.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size and/or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter and/or orientation of only a single eye410 (e.g., using only a single camera per pair of eyes) is determinedand assumed to be similar for both eyes of the user. The images obtainedby the inward-facing imaging system 466 may be analyzed to determine theuser's eye pose and/or mood, which can be used by the wearable system400 to decide which audio or visual content should be presented to theuser. The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IUs, accelerometers,gyroscopes, etc. The head's pose may be used alone or in combinationwith eye pose to interact with stem tracks and/or present audio content.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.In some cases, the user may use a finger (e.g., a thumb) to press orswipe on a touch-sensitive input device to provide input to the wearablesystem 400 (e.g., to provide user input to a user interface provided bythe wearable system 400). The user input device 466 may be held by theuser's hand during the use of the wearable system 400. The user inputdevice 466 can be in wired or wireless communication with the wearablesystem 400.

FIG. 5 shows an example of exit beams outputted by a waveguide. Onewaveguide is illustrated, but it will be appreciated that otherwaveguides in the waveguide assembly 480 may function similarly, wherethe waveguide assembly 480 includes multiple waveguides. Light 520 isinjected into the waveguide 432 b at the input edge 432 c of thewaveguide 432 b and propagates within the waveguide 432 b by TIR. Atpoints where the light 520 impinges on the DOE 432 a, a portion of thelight exits the waveguide as exit beams 510. The exit beams 510 areillustrated as substantially parallel but they may also be redirected topropagate to the eye 410 at an angle (e.g., forming divergent exitbeams), depending on the depth plane associated with the waveguide 432b. It will be appreciated that substantially parallel exit beams may beindicative of a waveguide with light extracting optical elements thatoutcouple light to form images that appear to be set on a depth plane ata large distance (e.g., optical infinity) from the eye 410. Otherwaveguides or other sets of light extracting optical elements may outputan exit beam pattern that is more divergent, which would require the eye410 to accommodate to a closer distance to bring it into focus on theretina and would be interpreted by the brain as light from a distancecloser to the eye 410 than optical infinity.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield. The optical system can include a waveguide apparatus, an opticalcoupler subsystem to optically couple light to or from the waveguideapparatus, and a control subsystem. The optical system can be used togenerate a multi-focal volumetric, image, or light field. The opticalsystem can include one or more primary planar waveguides 632 a (only oneis shown in FIG. 6 ) and one or more DOEs 632 b associated with each ofat least some of the primary waveguides 632 a. The planar waveguides 632b can be similar to the waveguides 432 b, 434 b, 436 b, 438 b, 440 bdiscussed with reference to FIG. 4 . The optical system may employ adistribution waveguide apparatus to relay light along a first axis(vertical or Y-axis in view of FIG. 6 ), and expand the light'seffective exit pupil along the first axis (e.g., Y-axis). Thedistribution waveguide apparatus, may, for example include adistribution planar waveguide 622 b and at least one DOE 622 a(illustrated by double dash-dot line) associated with the distributionplanar waveguide 622 b. The distribution planar waveguide 622 b may besimilar or identical in at least some respects to the primary planarwaveguide 632 b, having a different orientation therefrom. Likewise, atleast one DOE 622 a may be similar or identical in at least somerespects to the DOE 632 a. For example, the distribution planarwaveguide 622 b and/or DOE 622 a may be comprised of the same materialsas the primary planar waveguide 632 b and/or DOE 632 a, respectively.Embodiments of the optical display system 600 shown in FIG. 6 can beintegrated into the wearable system 200 shown in FIG. 2 .

The relayed and exit-pupil expanded light is optically coupled from thedistribution waveguide apparatus into the one or more primary planarwaveguides 632 b. The primary planar waveguide 632 b relays light alonga second axis, preferably orthogonal to first axis, (e.g., horizontal orX-axis in view of FIG. 6 ). Notably, the second axis can be anon-orthogonal axis to the first axis. The primary planar waveguide 632b expands the light's effective exit pupil along that second axis (e.g.,X-axis). For example, the distribution planar waveguide 622 b can relayand expand light along the vertical or Y-axis, and pass that light tothe primary planar waveguide 632 b which relays and expands light alongthe horizontal or X-axis.

The optical system may include one or more sources of colored light(e.g., red, green, and blue laser light) 610 which may be opticallycoupled into a proximal end of a single mode optical fiber 640. A distalend of the optical fiber 640 may be threaded or received through ahollow tube 8 of piezoelectric material. The distal end protrudes fromthe tube 642 as fixed-free flexible cantilever 644. The piezoelectrictube 642 can be associated with four quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outside,outer surface or outer periphery or diameter of the tube 642. A coreelectrode (not illustrated) is also located in a core, center, innerperiphery or inner diameter of the tube 642.

Drive electronics 650, for example electrically coupled via wires 660,drive opposing pairs of electrodes to bend the piezoelectric tube 642 intwo axes independently. The protruding distal tip of the optical fiber644 has mechanical modes of resonance. The frequencies of resonance candepend upon a diameter, length, and material properties of the opticalfiber 644. By vibrating the piezoelectric tube 8 near a first mode ofmechanical resonance of the fiber cantilever 644, the fiber cantilever644 is caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 644 is scanned biaxially in an area filling two dimensional(2-D) scan. By modulating an intensity of light source(s) 610 insynchrony with the scan of the fiber cantilever 644, light emerging fromthe fiber cantilever 644 forms an image. Descriptions of such a set upare provided in U.S. Patent Publication No. 2014/0003762, which isincorporated by reference herein in its entirety.

A component of an optical coupler subsystem collimates the lightemerging from the scanning fiber cantilever 644. The collimated light isreflected by mirrored surface 648 into the narrow distribution planarwaveguide 622 b which contains the at least one diffractive opticalelement (DOE) 622 a. The collimated light propagates vertically(relative to the view of FIG. 6 ) along the distribution planarwaveguide 622 b by total internal reflection (TIR), and in doing sorepeatedly intersects with the DOE 622 a. The DOE 622 a preferably has alow diffraction efficiency. This causes a fraction (e.g., 10%) of thelight to be diffracted toward an edge of the larger primary planarwaveguide 632 b at each point of intersection with the DOE 622 a, and afraction of the light to continue on its original trajectory down thelength of the distribution planar waveguide 622 b via TIR.

At each point of intersection with the DOE 622 a, additional light isdiffracted toward the entrance of the primary waveguide 632 b. Bydividing the incoming light into multiple outcoupled sets, the exitpupil of the light is expanded vertically by the DOE 4 in thedistribution planar waveguide 622 b. This vertically expanded lightcoupled out of distribution planar waveguide 622 b enters the edge ofthe primary planar waveguide 632 b.

Light entering primary waveguide 632 b propagates horizontally (relativeto the view of FIG. 6 ) along the primary waveguide 632 b via TIR. Asthe light intersects with DOE 632 a at multiple points as it propagateshorizontally along at least a portion of the length of the primarywaveguide 632 b via TTR. The DOE 632 a may advantageously be designed orconfigured to have a phase profile that is a summation of a lineardiffraction pattern and a radially symmetric diffractive pattern, toproduce both deflection and focusing of the light. The DOE 632 a mayadvantageously have a low diffraction efficiency (e.g., 10%), so thatonly a portion of the light of the beam is deflected toward the eye ofthe view with each intersection of the DOE 632 a while the rest of thelight continues to propagate through the primary waveguide 632 b viaTIR.

At each point of intersection between the propagating light and the DOE632 a, a fraction of the light is diffracted toward the adjacent face ofthe primary waveguide 632 b allowing the light to escape the TIR, andemerge from the face of the primary waveguide 632 b. In someembodiments, the radially symmetric diffraction pattern of the DOE 632 aadditionally imparts a focus level to the diffracted light, both shapingthe light wavefront (e.g., imparting a curvature) of the individual beamas well as steering the beam at an angle that matches the designed focuslevel.

Accordingly, these different pathways can cause the light to be coupledout of the primary planar waveguide 632 b by a multiplicity of DOEs 632a at different angles, focus levels, and/or yielding different fillpatterns at the exit pupil. Different fill patterns at the exit pupilcan be beneficially used to create a light field display with multipledepth planes. Each layer in the waveguide assembly or a set of layers(e.g., 3 layers) in the stack may be employed to generate a respectivecolor (e.g., red, blue, green). Thus, for example, a first set of threeadjacent layers may be employed to respectively produce red, blue andgreen light at a first focal depth. A second set of three adjacentlayers may be employed to respectively produce red, blue and green lightat a second focal depth. Multiple sets may be employed to generate afull 3D or 4D color image light field with various focal depths.

Other Components of the Wearable System

In many implementations, the AR system may include other components inaddition or in alternative to the components of the wearable systemdescribed above. The wearable system may, for example, include one ormore haptic devices or components. The haptic device(s) or component(s)may be operable to provide a tactile sensation to a user. For example,the haptic device(s) or component(s) may provide a tactile sensation ofpressure and/or texture when touching virtual content (e.g., virtualobjects, virtual tools, other virtual constructs). The tactile sensationmay replicate a feel of a physical object which a virtual objectrepresents, or may replicate a feel of an imagined object or character(e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the AR system. These physical objects may be referred to herein astotems. Some totems may take the form of inanimate objects, such as forexample, a piece of metal or plastic, a wall, a surface of table. Incertain implementations, the totems may not actually have any physicalinput structures (e.g., keys, triggers, joystick, trackball, rockerswitch). Instead, the totem may simply provide a physical surface, andthe AR system may render a user interface so as to appear to a user tobe on one or more surfaces of the totem. For example, the AR system mayrender an image of a computer keyboard and trackpad to appear to resideon one or more surfaces of a totem. For instance, the AR system mayrender a virtual computer keyboard and virtual trackpad to appear on asurface of a thin rectangular plate of aluminum which serves as a totem.The rectangular plate does not itself have any physical keys or trackpador sensors. However, the AR system may detect user manipulation orinteraction or touches with the rectangular plate as selections orinputs made via the virtual keyboard and/or virtual trackpad. The userinput device 466 (shown in FIG. 4 ) may be an embodiment of a totem may,which may include a trackpad, a touchpad, a trigger, a joystick, atrackball, a rocker switch, a mouse, a keyboard, amulti-degree-of-freedom controller, or another physical input device. Auser may use the totem, alone or in combination with poses, to interactwith the wearable system and/or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

Example Wearable Systems, Environments, and Interfaces

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 504, etc.) from one ormore user wearable systems (e.g., wearable system 200 and/or displaysystem 220) and/or stationary room systems (e.g., room cameras, etc.).The wearable systems can use various sensors (e.g., accelerometers,gyroscopes, temperature sensors, movement sensors, depth sensors, GPSsensors, inward-facing imaging system, outward-facing imaging system,etc.) to determine the location and various other attributes of theenvironment of the user. This information may further be supplementedwith information from stationary cameras in the room that may provideimages and/or various cues from a different point of view. The imagedata acquired by the cameras (such as the room cameras and/or thecameras of the outward-facing imaging system) may be reduced to a set ofmapping points.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize and/or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects andsupplement this with semantic information to give life to the objects.For example, if the object recognizer recognizes a set of points to be adoor, the system may attach some semantic information (e.g., the doorhas a hinge and has a 90 degree movement about the hinge). If the objectrecognizer recognizes a set of points to be a mirror, the system mayattach semantic information that the mirror has a reflective surfacethat can reflect images of objects in the room. Over time the mapdatabase grows as the system (which may reside locally or may beaccessible through a wireless network) accumulates more data from theworld. Once the objects are recognized, the information may betransmitted to one or more wearable systems. For example, the MRenvironment 700 may include information about a scene happening inCalifornia. The environment 700 may be transmitted to one or more usersin New York. Based on data received from an FOV camera and other inputs,the object recognizers and other software components can map the pointscollected from the various images, recognize objects etc., such that thescene may be accurately “passed over” to a second user, who may be in adifferent part of the world. The environment 700 may also use atopological map for localization purposes.

As another example, a user's environment may be associated with certainaudio content. For example, the stem tracks of an audio file may beassociated with different locations of a user's room. The wearablesystem may pass the map of the user's environment together with thelocation information of stem tracks to another user thereby allowing theother user can interact with the stem tracks. As a result, the two userscan collaborate on creating new audio files by mixing the stem tracks inthe user's room.

As yet another example, a user can interact with stem tracks in hisenvironment to create a new audio file. The wearable system of the usercan pass the new audio files to other wearable systems for play back orbroadcast the new audio file to other people (for example via speakers).

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be represented to a user of the MRsystem (e.g., a wearable system). The user may be geographically remotefrom the scene. For example, the user may be New York, but may want toview a scene that is presently going on in California, or may want to goon a walk with a friend who resides in California.

At block 810, the AR system may receive input from the user and otherusers regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, and/or hand gestures) that can be used in displaying andunderstanding the orientation and position of various objects in theuser's surroundings. The object recognizers 708 a, 708 n may crawlthrough these collected points and recognize one or more objects using amap database at block 830. This information may then be conveyed to theuser's individual wearable system at block 840, and the desired virtualscene may be accordingly displayed to the user at block 850. Forexample, the desired virtual scene (e.g., user in CA) may be displayedat the appropriate orientation, position, etc., in relation to thevarious objects and other surroundings of the user in New York.

FIG. 9 is a block diagram of another example of a wearable system. Inthis example, the wearable system 900 comprises a map, which may includemap data for the world. The map may partly reside locally on thewearable system, and may partly reside at networked storage locationsaccessible by wired or wireless network (e.g., in a cloud system). Apose process 910 may be executed on the wearable computing architecture(e.g., processing module 260 or controller 460) and utilize data fromthe map to determine position and orientation of the wearable computinghardware or user. Pose data may be computed from data collected on thefly as the user is experiencing the system and operating in the world.The data may comprise images, data from sensors (such as inertialmeasurement devices, which generally comprise accelerometer andgyroscope components) and surface information pertinent to objects inthe real or virtual environment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (SLAM or V-SLAM, referring to a configurationwherein the input is images/visual only) process. The system can beconfigured to not only find out where in the world the variouscomponents are, but what the world is made of Pose may be a buildingblock that achieves many goals, including populating the map and usingthe data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 940, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors) may serve as input to the Stereo process 940. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces are efficient (e.g., relativeto a large point cloud) and digestible inputs to other processingdevices like game engines. Thus, the output of the Stereo process (e.g.,a depth map) 940 may be combined in the Fusion process 930. Pose may bean input to this Fusion process 930 as well, and the output of Fusion930 becomes an input to populating the map process 920. Sub-surfaces mayconnect with each other, such as in topographical mapping, to formlarger surfaces, and the map becomes a large hybrid of points andsurfaces.

To resolve various aspects in a mixed reality process 960, variousinputs may be utilized. For example, in the embodiment depicted in FIG.9 , Game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding where such objects are relative to each other, tobe another valuable input to mixed reality. Pose relative to the worldbecomes an input as well and plays a key role to almost any interactivesystem.

Controls or inputs from the user are another input to the wearablesystem 900. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 900 regarding what he or she wants to do. Beyond justmoving oneself in space, there are various forms of user controls thatmay be utilized. In one embodiment, a totem, user input device, orobject such as a toy gun may be held by the user and tracked by thesystem. The system preferably will be configured to know that the useris holding the item and understand what kind of interaction the user ishaving with the item (e.g., if the totem or object is a gun, the systemmay be configured to understand location and orientation, as well aswhether the user is clicking a trigger or other sensed button or elementwhich may be equipped with a sensor, such as an IMU, which may assist indetermining what is going on, even when such activity is not within thefield of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 900 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 900 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.

Eye tracking is another input (e.g., tracking where the user is lookingto control the display technology to render at a specific depth orrange). In one embodiment, vergence of the eyes may be determined usingtriangulation, and then using a vergence/accommodation model developedfor that particular person, accommodation may be determined.

With regard to the camera systems, the example wearable system 900 shownin FIG. 9 can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle theStereo imaging process 940 and also to capture hand gestures andtotem/object tracking in front of the user's face. The cameras in thethree pairs of cameras may be a part of the outward-facing imagingsystem 464 (shown in FIG. 4 ). The wearable system 900 can include eyetracking cameras (which may be a part of an inward-facing imaging system462 shown in FIG. 4 ) oriented toward the eyes of the user in order totriangulate eye vectors and other information. The wearable system 900may also comprise one or more textured light projectors (such asinfrared (IR) projectors) to inject texture into a scene.

FIG. 10 is a process flow diagram of an example of a method 1000 fordetermining user input to a wearable system. In this example, the usermay interact with a totem. The user may have multiple totems. Forexample, the user may have designated one totem for a social mediaapplication, another totem for playing games, etc. At block 1010, thewearable system may detect a motion of a totem. The movement of thetotem may be recognized through the user's FOV camera or may be detectedthrough sensors (e.g., haptic glove, image sensors, hand trackingdevices, eye-tracking cameras, head pose sensors, etc.).

Based at least partly on the detected gesture, eye pose, head pose, orinput through the totem, the wearable system detects a position,orientation, and/or movement of the totem (or the user's eyes or head orgestures) with respect to a reference frame, at block 1020. Thereference frame may be a set of map points based on which the wearablesystem translates the movement of the totem (or the user) to an actionor command. At block 1030, the user's interaction with the totem ismapped. Based on the mapping of the user interaction with respect to thereference frame 1020, the system determines the user input at block1040.

For example, the user may move a totem or physical object back and forthto signify turning a virtual page and moving on to a next page or movingfrom one user interface (UI) display screen to another UI screen. Asanother example, the user may move their head or eyes to look atdifferent real or virtual objects in the user's FOR. If the user's gazeat a particular real or virtual object is longer than a threshold time,the real or virtual object may be selected as the user input. In someimplementations, the vergence of the user's eyes can be tracked and anaccommodation/vergence model can be used to determine the accommodationstate of the user's eyes, which provides information on a depth plane onwhich the user is focusing. In some implementations, the wearable systemcan use raycasting techniques to determine which real or virtual objectsare along the direction of the user's head pose or eye pose. In variousimplementations, the ray casting techniques can include casting thin,pencil rays with substantially little transverse width or casting rayswith substantial transverse width (e.g., cones or frustums).

The user interface may be projected by the display system as describedherein (such as the display 220 in FIG. 2 ). It may also be displayedusing a variety of other techniques such as one or more projectors. Theprojectors may project images onto a physical object such as a canvas ora globe. Interactions with user interface may be tracked using one ormore cameras external to the system or part of the system (such as,e.g., using the inward-facing imaging system 462 or the outward-facingimaging system 464).

FIG. 11 is a process flow diagram of an example of a method 1100 forinteracting with a virtual user interface. The method 1100 may beperformed by the wearable system described herein.

At block 1110, the wearable system may identify a particular UI. Thetype of UI may be predetermined by the user. The wearable system mayidentify that a particular UI needs to be populated based on a userinput (e.g., gesture, visual data, audio data, sensory data, directcommand, etc.). At block 1120, the wearable system may generate data forthe virtual UI. For example, data associated with the confines, generalstructure, shape of the UI etc., may be generated. In addition, thewearable system may determine map coordinates of the user's physicallocation so that the wearable system can display the UI in relation tothe user's physical location. For example, if the UI is body centric,the wearable system may determine the coordinates of the user's physicalstance, head pose, or eye pose such that a ring UI can be displayedaround the user or a planar UI can be displayed on a wall or in front ofthe user. If the UI is hand centric, the map coordinates of the user'shands may be determined. These map points may be derived through datareceived through the FOV cameras, sensory input, or any other type ofcollected data. As an example, the wearable system can generategraphical representations of various stem tracks in an audio file aroundthe user.

At block 1130, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 1140, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 1150. Forexample, the UI may be a body centric ring around the user's body. Thewearable system may then wait for the command (a gesture, a head or eyemovement, input from a user input device, etc.), and if it is recognized(block 1160), virtual content associated with the command may bedisplayed to the user (block 1170). As an example, the wearable systemmay wait for user's hand gestures before mixing multiple steam tracks.

Additional examples of AR systems, UI, and user experiences (UX) aredescribed in U.S. Patent Publication No. 2015/0016777, which isincorporated by reference herein in its entirety.

Examples of Stem Tracks

As described with references to FIGS. 2 and 4 , a wearable system canplay an audio file and present visualizations of audio content to auser. The wearable system can be part of a wearable device (such as ahead-mounted device/display) configured to present AR, VR, and/or MRenvironment. The audio file can be stored and processed using localprocessing and data module 260, remote processing module 270, or remotedata repository 280, in combination or the like.

An audio file can include a plurality of tracks. For example, the audiofile can include a stereo master track, tracks for the “right” or “left”channels of a stereo system, tracks for surround sound channels, and soforth. Certain types of audio files can include a plurality of tracksrepresenting different instruments (e.g., guitar, bass, drums, piano,synthesizer, horns) or vocals (e.g., lead vocal, backup vocal). Tracksrepresenting different instruments or vocals are sometimes referred toas stem tracks or stems. In various implementations, the number of stemtracks can be one, two, three, four, five, six, seven, eight, ten,twelve, or more.

Stem tracks can be any partition of an audio file, where each stem trackrepresents a different audio content of the audio file. As an example, astem track can represent a particular instrument or vocal (e.g., guitaror lead vocal) in a recording. For example, a recording of chamber music(e.g., a string quartet) can include four separate stem tracks whichrepresent the first violin, the second violin, the viola, and the cello.As another example, a stem track can represent a group of instruments orvocals (e.g., a rhythm section of bass and drums or a chorus). Forexample, for an orchestra, there may be different stem tracksrepresenting the violins, the wood winds, the percussion, the brass, thebass, etc. Stem tracks are not limited to musical recordings and can beused for recordings of speech (e.g., different stems for differentspeakers).

Stem tracks can be associated with spatial locations in a user's FOR. Asa result, the sound of the stem track may audibly appear to come fromits associated spatial location. For example, where a stem track isassociated with the right side of the user, the sound of the stem trackmay appear to come from the right side of the user. As the user changeshis pose, the user can perceive a different sound field or visual field.In the same example, if the user turns 180 degrees, the sound may appearto come from the left side (instead of the right side) of the user. Thisadvantageously provides a unique, user-controllable user experience.

A stem track's location can be defined relative to the user (e.g., theuser's head), such as at a certain distance and/or angle with respect tothe user. Additionally or alternatively, the stem track's location canbe defined globally. As an example, a stem track may be positioned to beon the front wall while another stem track is positioned to be in themiddle of the user's room.

The location of the stem track may change over time. For example, a stemtrack may be associated with a singer's voice. The position of the stemtrack may change as if the singer is (virtually) moving around in theuser's environment. The location of the stem track may also be changedbased on user interactions. As further described below, a user can movea stem track, for example, by grabbing the visual graphic associatedwith the stem track.

The wearable system can provide visualizations of individual stem tracksand/or one or more stem tracks combined. Thus, the wearable system canadvantageously permit the user to visualize the audio spatialization andinteract with the stem tracks.

In some implementations, the audio file may include the stem tracks plusadditional information (e.g., metadata) that can be used by the wearabledevice to render an audio or visual representation of the music. Forexample, the additional information may include information relating tomovements of musical performers on a stage (e.g., a lead singer movingor dancing around the stage), information about the acoustics or size ofthe performance venue (e.g., the size of the stage, the size of theconcert hall, etc.), and so on. The wearable system can use thisadditional information to present a visual representation of aperformance by a musical artist. For example, the movement informationfor a lead singer can be used to determine where, in the user's visualfield, a representation (e.g., image or avatar) of the lead singershould be displayed as the lead singer moves around during theperformance. Such embodiments can be used to create a visual renderingof a live performance where the audio track was recorded.

The information on audio stems, time series positional data for thestems, visualizations of the stem tracks, as well as one or more of theadditional pieces of information may be parsed from the metadata of theaudio file. For example, the audio file may be encoded in a format whichsupports such metadata. The metadata may specify the stem tracks andvisualizations (such as shape, animation, etc.) of the associated stemtracks of the audio file. The metadata can also specify active stemtracks as well as the positions of the stem tracks at a given time ofthe audio file. The audio file may be in a proprietary format. To allowthe audio file to be used with audio equipment that might not have thefunctionality to provide visualizations and mixing of stems as disclosedherein (e.g., to provide backward compatibility), the audio format maystore such metadata so that older audio systems can ignore the metadata(e.g., by not reading it from the file) and read only the stereo mix ofthe sound file, for example.

Examples of Mixing Stem Tracks

Two or more stem tracks can be combined to produce an audio mix. Theaudio mix may simulate a sound field which reflects the spatiallocations of the stems tracks. The audio mix may also reflect a mixtureof the stem tracks, regardless of their spatial locations.

The combination (or mixing) of the stem tracks for the differentinstruments or vocals produces the audio mix that can be presented to auser for playback by the wearable system (e.g., via the speaker 240).The audio mix can be used to produce one or more audio channels (e.g.,stereo or surround sound (which can have 5, 7, or more channels)). Thewearable system can also modify the audio mix, such as e.g., the stemtracks level (e.g., loudness or softness), frequency content, dynamics(e.g., style (staccato or legato), speed), panoramic position (sometimesreferred to as panning, e.g., distribution of the musical sounds amongthe stereo or multi-channel sound field), and can add effects (e.g.,reverb), in combination or the like.

The user can interact with the stem tracks using poses or a user inputdevice, alone or in combination. As one example, the spatialization orvisualization of the mix can be based (at least partly) on the user'shand movements or by control signals received from the user input device466 (e.g., mouse or joy stick, controllable by the user or a thirdparty). The wearable system can identify a position of the user's hands.As the hands move, the wearable system can generate a different mix ofthe stem tracks for presentation (audibly or visually) to the user. Insome implementations, the user may use his or her hand (or stylus orbaton) to “touch” or “move” a real or virtual object seen through thedisplay 220 of the wearable system and the wearable system may modify oradjust the mix in response to the touch or move. As another example, theuser may identify a stem track (e.g., using the user input device) andmove the stem track along a trajectory using his arm. The stem track maycollide with another stem track. The wearable system may present a mixof these two stem tracks to the user.

In some implementations, while the user is listening to an audio file,the wearable system can dynamically mix the stem tracks of the audiofile based on the user's pose (such as head, eye, foot, hand gestures,or other body poses). For example, as the user's eye(s) or head moves,the wearable system (e.g., via the processing modules 260 and/or 270)can dynamically mix the stem tracks in the audio file to create a “new”set of sounds for presentation to the user (e.g., via the speaker 240).Thus, the wearable system can advantageously spatialize the mix of thestem tracks of the audio file.

In addition, the wearable system (e.g., via the processing modules 260and/or 270) can dynamically present visualizations of individual stemtracks or visualizations of mixed stem tracks to the user (e.g., via thedisplay 200) based on the user's pose. This allows the user to seedifferent visual representations of the audio mix of the stem tracks asthe user's head or eye(s) pose changes.

In some implementations, the audio file may include the stem tracks aswell as information associated with one or more pre-mixed combinationsof the stem tracks. In such implementations, rather than dynamicallymixing the stem tracks as the user's pose changes, the wearable systemmay access one of the pre-mixed combinations for presentation to theuser. The wearable system can select a pre-mixed combination based onthe user's position, the user's pose, input from the user input device,etc., alone or in combination. For example, the pre-mixed combinationmay reproduce the music as if it were circling around the user's head.

The user of the wearable system can also “lock” the position of soundsources in an environment (such as the user's room) or relative to anobject (such as the user's head or body). For example, the user may movethe stem tracks in his environment. Once the user finds an audio mix helikes, he can lock the spatial location of the stem track in the audiomix, for example, by actuating a user input device or by his pose. Theaudio mix can accordingly remain the same, regardless of the user'sposition in his environment. The user can store the “locked” mix in anaudio file so that the user can re-experience the visual and soundfields in the future by replaying the audio file.

Example User Interfaces for Presenting Stem Tracks

FIGS. 12, 13, and 14 schematically illustrate examples of userinterfaces which present to a user of a wearable system visualizationsof multiple stem tracks of an audio file. The stem tracks may bedynamically mixed based at least in part on the user's pose.

FIG. 12 schematically illustrates two example user interfaces 1210 and1220. In the example interfaces 1210 and 1220, a mix of five stem tracksis shown. In the example user interface 1210, five different visualgraphics (e.g., icons or orbs) 1210 a-1210 e are shown as representingthe corresponding five stem tracks. The visual graphics need not bestatic and can move or be modified or altered as the audio file isplayed. In this example, when the user's head or eye pose changes, theaudio mix changes and the visual representation 1210 correspondinglychanges. For example, the user's eye (or head) pose may indicate theuser is looking toward the direction of the graphic 1210 a. In some suchcases, the graphic 1210 a may be displayed as emphasized (e.g., larger,brighter, etc.) than the graphics 1210 b-1210 e that are not in theuser's gaze direction. The sound associated with the graphic 1210 a canalso be emphasized, for example, by an increase in volume. As the user'spose changes, another visual graphic may be emphasized or the user maybe presented with a different sound field (corresponding to thedifferent audio mix of the stem tracks) and a different visual field.

In some implementations, the wearable system may determine whether toemphasize a visual graphic based on the content of the audio file. Forexample, the wearable system may emphasize the visual graphic 1210 abecause its corresponding stem track is emphasized in volume or is thedominate track in the audio file.

In example user interface 1220, the visual graphics 1220 a and 1220 care presented as Lissajous figures. In this example, there may be 5 stemtracks (1220 a through 1220 e) in the audio file in the user's FOV,although there may be additional stem tracks in the user's environment.However, the stem tracks corresponding to the visual graphics 1220 b,1220 d, and 1220 e are muted. As a result, the wearable system does notshow (or greatly reduces in size or brightness) visual graphics in thesedirections (the points 1220 b, 1220 d, and 1220 e are shown as dots toindicate that no (or a reduced) graphic is displayed to the user). Theuser will hear (e.g., through speaker 240) an audio mix of just the stemtracks associated with directions 1220 a, 1220 c.

In various implementations, the visual graphics 1210 a-1210 e, 1220a-1220 e can be different from or the same as each other. Many types ofvisual graphics can be presented (see, e.g., the examples in FIGS. 13and 14 ), and the examples described herein are intended to beillustrative and not limiting. For example, the Lissajous figures 1220a, 1220 c illustrated in FIG. 12A can be dynamic and their patterns canchange in time, e.g., with the beat of the music.

In some implementations, the wearable system may use (at least in part)audio spectral (frequency) content of the stem tracks to determine thevisualization of a corresponding visual graphic in the visual field ofthe user. The audio spectral content may be used to determine theplacement of the visual graphic. For example, high frequencies may bedisplayed in a first direction, and low frequencies may be displayed ina different, second direction. As another example, sounds with differentspectral content can be represented by visual graphics in differentportions of the user's FOR (including in directions for which the userwould have to turn around to see or hear). The audio spectral contentmay also be used to determine the appearance of the visual graphic. Forexample, wearable system can generate Lissajous figures corresponding toa stem track based on the audio spectral content of that stem track.

The user interfaces 1210 and 1220 may be presented in a VR mode ofdisplay, where the user sees the visual graphics 1210 a-1210 e or 1220a-1220 e, but not the outside world 470 (shown in FIG. 4 ). The userinterfaces 1210 and 1220 may also be presented in an AR or an MR mode ofdisplay, where the user sees the visual graphics 1210 a-1210 e or 1220a-1220 e superimposed on the outside world 470 (shown in FIG. 4 ).

Example Interactions with Stem Tracks

The user can alter individual sound characteristics of a stem track. Asan example of user interaction with the user interfaces 1210, 1220 in anAR mode of display, a disc jockey (DJ) can use hand gestures to controlindividual sound characteristics (e.g., volume) of the stem tracks, asthey are displayed by a wearable system. A DJ utilizing the wearablesystem may “grab” one of the visual graphics, such as the visual graphic1210 a, to control the volume of that stem track. If the visual graphic1210 a represents the vocals track, the DJ may compress his hand toreduce the volume of that stem track. Other sound characteristics of thestem track may be altered or modified. For example, if the DJ were toselect a different sound characteristic mode (e.g., by making some otherhand gesture, foot pose), the DJ could alter or modify another soundcharacteristic of the vocals track represented by the visual graphic1210 a. If the sound characteristic mode is changed to pitch, the DJcompressing her hand in this case would decrease the pitch. If, on theother hand, the DJ wanted to increase the pitch, the DJ may use anotherhand gesture to increase the pitch associated with the visual graphic1210 a. For example, the DJ may “spin” the graphic 1210 a by moving herhand quickly in either a clockwise or counterclockwise passion in anarea associated with the graphic 1210 a (e.g., on or near the graphic).

Other sound characteristics that may alter or modify the various stemtracks are possible. For example, other sound characteristics caninclude volume, pitch, bass, treble, tonal quality, intensity, duration,etc. Although the foregoing example has been described in the context ofhand gestures, other body gestures can be used to control the soundcharacteristics of stem tracks appearing on the display of a wearablesystem.

Additionally or alternatively, some stem tracks may be displayed in a VRmode. Poses can also be used to alter or modify the stem tracksdisplayed in the VR mode. As another example, a DJ may tap one of theirfeet twice (e.g., tapping a foot twice within a certain time period of afirst tap) to indicate that a sound characteristic of a stem track is tobe altered or modified. Generally, various embodiments of gestures maybe used to interact with the visual display. While the foregoing examplehas been described in terms of poses, control signals from a user inputdevice may also alter stem tracks. For example, a user may utilize ajoystick, totem, or touch-sensitive input device to alter the pitch of aparticular stem track being represented as a visual graphic.

A stem track or portion thereof may also be generated by the user of thewearable system. A user generated stem track can replace thecorresponding one (or more) stem track(s) of an audio file. For example,the user can substitute the user generated stem track for one of theaudio's original stem tracks. The user can record a vocal track (of theuser or someone else) and substitute the user vocal track for theoriginal vocal track of the recording. Thus, the stem tracks beingplayed may sound as if the user is singing on the audio recording. Inanother embodiment, if two stem tracks correspond to respective vocaltracks of a duet, a user generated vocal stem track can replace one ofthe voices in a duet. As yet another example, a user generated guitarsolo can replace the corresponding guitar stem track of an audiorecording. Any number of the audio stem tracks can be replaced, in wholeor in part, with user generated stem tracks.

FIG. 13 schematically illustrates an example of an MR presentation 1300of an audio file comprising stem tracks. In this example, the user looksthrough the display 220 (shown in FIG. 2 ) out into a room 1310 thatcontains a real (physical) object such as a sofa 1312. Only one physicalobject is shown in FIG. 13 for illustrative purpose but this is not alimitation.

In this example, the audio file includes the stem tracks that correspondto vocals, guitar, bass, and drums. The wearable system presents avisual graphic 1322 in the form of a person playing a guitar for theguitar track, a visual graphic 1328 in the form of a person playing abass guitar for the bass track, a visual graphic 1326 in the form of aperson singing for the vocal track, and a visual graphic 1324 in theform of a person playing a drum set for the drum track. Similar to theuser interfaces 1210 and 1220 shown in FIG. 12 , the different visualgraphics 1322, 1324, 1326, and 1328 are displayed in differentdirections in the room 1310. In this MR presentation, the visual graphic1322 for the guitar player is presented as if the guitar player weresitting on the (physical) sofa 1312 that is actually present in the room1310. The visual graphics 1324, 1326, and 1328 are presented in regionsof the room where there are no physical objects.

Since embodiments of the wearable system described herein are capable ofpresenting objects at multiple depth planes in the user's visual field(see, e.g., the discussion of FIG. 3 , the waveguides 432 b, 434 b, 436b, 438 b, 440 b in the waveguide assembly 480 of the display system 400of FIG. 4 , or the waveguide 632 b of the display system 600 of FIG. 6), the user interface 1300 shows the different visual graphics 1322,1326, 1324, and 1328 at different depths in the room. For example, thebass guitarist (the visual graphic 1328) is toward the front of the room1310, the lead singer (the visual graphic 1326) is farther back in theroom 1310 but forward of the drummer (the visual graphic 1324), who isat the back of the room, farthest from the user. Additionally, in thisexample, the amount of detail in the different visual graphics candepend on depth. For example, since the drummer (the visual graphic1324) appears as being farthest from the user, the visual graphic 1324has less detail than the visual graphics 1326 and 1328, which appear ascloser to the user. In embodiments where the audio file includesadditional information relating to movement (e.g., of the lead singer),the wearable system can present the visual graphic 1326 representing thesinger as moving around in the room 1310, for example.

The example user interface 1300 illustrated in FIG. 13 accordingly showsan example in which the different stem tracks are represented visuallyin the external world environment 470 (shown in FIG. 4 ) of the user.The wearable system may recognize objects in the environment 470 (e.g.,the sofa 1312 in the room 1310), for example, by analyzing the imagesacquired by the outward-facing imaging system or may be in communicationwith totems or electronic trackers disposed in the environment 470 thatcan be used to assist in the display of the visual graphics. Forexample, a visual graphic for a particular stem track may be visualizedat the position of a totem in the room 1310.

As the user moves around in the room 1310, the presentations of the stemtracks may change accordingly. For example, when the user walks close tothe back of the room, the wearable system can present the visual graphic1324 in more detail. The wearable system can also increase the volume ofthe drums track associated with the visual graphic 1324 to reflect thatthe user is closer to the drummer. In some implementations, the wearablesystem can adjust spatializations of stem tracks based on the locationof the user. For example, when the visual graphic 1326 is presented infront of the user, the wearable system may play the audio file as if thevocal track is from the front of the user. However, as the user walkspass the visual graphic 1326 and towards the visual graphic 1324, thewearable system may play the audio file as if the vocal track is comingfrom the right side of the user.

In certain implementations, the wearable system can update the locationsof the visual graphics as the user walks around. For example, withreference to FIG. 13 , as the user moves around, the visual graphics1322, 1324, 1326, and/or 1328 may “follow” the user in the room 1310.

The user of the wearable system can selectively choose a desired mix ofthe audio tracks. For example, if the stem tracks correspond to vocals,guitar, bass, and drums, the user could listen to (or see a visualrepresentation of) just one of the stem tracks (e.g., just the vocals orjust the guitar). The wearable system may present (visually and audibly)to the user only the stem tracks in the user's FOV. As the user moveshis or her head or eyes, the stem tracks may move inside or out of theuser's FOV. Accordingly, the user can listen to (or see) a differentstem track or a different mix of the stem tracks. In someimplementations, the user can mute an audio track by using anappropriate gesture (e.g., a hand gesture). For example, the user maywant to listen to a guitar solo and might mute all tracks except theguitar track.

Additionally or alternatively, as described above with respect to theexample of interacting with the visual display to control a soundcharacteristic, the visual graphics may also be modified or altered in aVR, AR, and/or MR mode of display using hand gestures. As one example,in an MR mode of display, a user may depress her hand on the bassguitarist, e.g. the visual graphic 1328, to depress the bass associatedwith the sound of the soundtrack played by the visual graphics 1322,1326, 1324, and 1328. The resulting display of the visual graphic 1328may appear smaller than the other graphics 1322, 1326, or 1324. Asanother example, the user can use hand gestures to control the volume ofthe stem track represented by visual graphics 1322, 1326, 1324, and1328. The user may “grab” and “drag” the visual graphic 1324 of thedrummer toward the front of the room 1310 to increase the volume of thedrums. Conversely, to decrease the volume of the vocals, a user may“push back” the visual graphic 1324 of the drummer towards the back ofthe room 1310.

The visual graphics may also be modified or altered using other poses.For example, other sensors coupled to the wearable system 200, such asinertial measurement units (IMUs), may be used to detect head or bodypose of the user. For example, the IMUs may detect the user nodding herhead backwards and forward, which may correspond to a change in headpose. The wearable system can use this change in head pose for selectionof one of the visual graphics 1322, 1324, 1326, and 1328. For example,while the user is looking at the visual graphic 1328 of the bass andnods her head, the wearable system can select the stem trackcorresponding to the bass. In some cases, a sufficiently “sharp” headmovement (e.g., IMU measurement above a threshold) may indicate aselection of the visual graphic, whereas a more gentle head movement(IMU measurement below a threshold) may indicate adjustment of aproperty associated with the graphic. For example, while the user is“pushing back” the visual graphic 1326 towards the back of the room1310, the user may nod her head towards the visual graphic 1328 toindicate the selection of the visual graphic 1328, e.g., the bass stemtrack. Once selected, the user can alter that stem track in some way(e.g., dampen the bass stem track), or as described above with respectto changes of the sounds characteristics of stem tracks.

Additionally or alternatively, while altering a selected stem track withposes, the user may also perform the particular stem track and replacethat corresponding stem track for the audio track being played by thewearable system (e.g., as described above with respect to the example ofreplacing stem tracks). For example, the user may replace the vocals ofthe graphic 1326, while simultaneously altering other stem tracksrepresented by the other visual graphics, e.g., the visual graphic 1328representing the bass stem track. Further, body poses besides head poseor hand gestures can be used. For example, a change in foot pose (e.g.,tapping a foot) may indicate stepping on a guitar or drum pedal, hittinga drum, actuating an audio effect (e.g., reverb), and so forth.

FIG. 14 schematically illustrates another example of a user interface1400 that can be presented by a wearable system. In this example, theaudio file includes a string quartet (two violins, one viola, and onecello) playing a piece of chamber music. As the user of the wearablesystem listens to the audio file, the wearable system presents the userinterface 1400 to the user and shows visual graphics 1410 and 1420representing the violins, a visual graphic 1430 representing the cello,and a visual graphic 1440 representing the viola. As the user's head oreye pose changes, a different mix of these four instruments is presentedaudibly to the user. For example, if the user looks toward the directionof the violins 1410 and 1420, the user may hear audio in which theviolin sounds are emphasized over the cello and viola sounds (which maybe muted or played at lower volumes than the violins). The visualgraphics 1410 and 1420 for the violins may correspondingly be emphasizedwith respect to the visual graphics 1430 and 1440 for the cello andviola. For example, the bows of the violins may be displayed as movingwhile the bows for the cello and viola may be shown as not moving.Although in this example, the visual graphics 1410, 1420, 1430, and 1440appear as displayed at a common depth plane, this is not a limitationand in other example, the visual graphics representing the differentinstruments can be presented at different depth planes from the user.Additional examples of such 3D representation are shown in FIG. 15 .

The wearable system may also be configured to allow user interactionswith the displayed visual graphics. For example, the user interface 1400may present various stem tracks with which the user can interact in theparticular fashion that stem track is being represented or displayed. Aparticular visual graphic may be a representation of what may be alteredor modified. As an example, in an MR mode of display, if the user“grabs” the viola 1440 (e.g., “grabs” the viola as described above withrespect to the example of interacting with graphics displayed in the MRmode), the user may “play” the viola to modify a characteristic of theviola sounds. Continuing in the example, after the user “grabs” theviola, the user may use a wearable system generated “bow” to “play” theviola, thereby modifying a characteristic of a viola sound (e.g.,volume). As another example of an MR mode of display, the user may“play” the viola to replace the corresponding viola stem track beingplayed. For example, the user may use a wearable system generated “bow”to play the viola and generate a viola stem track. In one embodiment,this user generated viola stem track may replace the viola stem track instring quartet, so that the resulting audible sounds to the user is theuser playing the viola portion of a string quartet.

Although the foregoing example has been described in the context ofvisual graphics of chamber instruments, other visual graphics may act asa UI to alter or modify the object for which the generated UIrepresents. For example, for audio tracks with electric guitars, the UImay include a representation of an electric guitar having strings,switches, and knobs. The user may pluck or strum the guitar strings,adjust the toggle switches, or rotate the knobs to control variousaspects of the audio track. The visual graphic may be a skeuomorphicrepresentation of the object to be modified. For example, a telephonegraphic including a number keypad may represent a dialing functionassociated with the wearable system so that the user can dial atelephone number. As another example, an AR representation of a calendarmay emulate the appearance of binding on a paper desk calendar.Generally speaking, any visual graphic object generated by a wearablemay be used as a user interface.

Presenting Stem Tracks in the 3D Space

As described with reference to FIG. 13 , the wearable system can presenta 3D user interface which shows visual graphics at different depths inthe user's environment. FIG. 15 illustrates another example of the 3Duser interface.

In FIG. 15 , there are 4 visual graphics 1510, 1520, 1530, and 1540 inthe user's FOV. These visual graphics may be Lissajous figures or othergraphical representations. The appearance of the visual graphics cancorrespond to the audio content being played to the user. For example,the user may perceive the visual graphic 1510 as the being the closestto him because its associated stem track is emphasized in the audio file(e.g., played louder in the mix). The visual graphic 1520 is shown in areduced size because its associated stem track may be relatively muted.The visual graphics may change as the audio content changes. Forexample, when the stem track associated with the visual graphic 1520 isno longer muted, visual graphic 1520 may no longer appear in the reducedsize.

The wearable system can place a visual graphic based on the spatiallocation of its associated stem. For example, the visual graphic 1520may appear to be small because its corresponding sound comes from afaraway location (e.g., a timpanist at the back of an orchestra) whereasthe visual graphic 1510 may appear larger because its correspondingsound comes from a more nearby location (e.g., a singer in front of theorchestra). As another example, the visual graphic 1540 appears to bebehind the visual graphic 1530 because the stem track associated withthe visual graphic 1540 may be further away from the user than the stemtrack associated with the visual graphic 1530. The relative distances orlocations of the graphics 1510-1540 can be displayed to provide a 3Dexperience for the user and may dynamically change (in distance,location, size, color, shape, etc.) as the music changes or as theuser's pose changes. The graphics 1510-1540 may be interactable suchthat the user can select one or more of them (e.g., by reaching out withthe user's hand) and move the selected graphic around, which may alter aproperty of not only the visualization but also the sound (e.g., bymaking the stem track corresponding to the selected graphic louder,softer, have more or less bass or treble, etc.).

The wearable system can also move the visual graphics out of the user'sFOV. FIG. 15 shows a horizon 1550 in the 3D view 1500. The wearablesystem can show that virtual graphics submerge under or rise from thehorizon 1550. As an example, when the volume of a stem track isdecreasing, the wearable system may show its associated visual graphicmoving underneath the horizon 1550. As a result, the user may not beable to see the portion of the visual graphic underneath the horizon1550. As another example, when the volume of the stem track increases,the wearable system may show that the associated visual graphic risesfrom the horizon 1550. In some implementations, the wearable system mayshow the horizon as a reflective surface (such as simulating watersurface or a glass) where wearable system can show reflected images ofthe visual graphics 1510, 1520, 1530 and 1540 on the horizon 1550.

In some embodiment, the wearable system may display a visual graphiconly when a user is sufficiently close the visual graphic. For example,in FIG. 15 , the system may be configured not to show the visual graphic1520 because it is too far away from the user. As further described withreference to FIG. 16B, wearable system may be configured to only displaythe visual graphic when the user can also hear the sound sourceassociated with the visual graphic, such as, for example, when the useris inside of the audible regions of a directional sound source.

As another example, FIG. 16A shows a bird's eye view 1600 a of a user'senvironment. While the wearable system is playing a piece of music, thewearable system may present five (in this example) visual graphics 1610a, 1620 a, 1630 a, 1640 a, and 1650 a, each associated with a stem trackof the music. In this example, the five visual graphics are placed atdifferent locations of the user's environment. The user 1670 a may standin the middle of the room and be surrounded by the five visual graphics.At a given time, the user 1670 a may see a subset of the visual graphicsin his FOV even though there are other visual graphics in hisenvironment. For example, the user 1670 a may see visual graphics 1630 aand 1640 a at the same time in his FOV, but not the visual graphic 1610a because the visual graphic 1610 a is behind the user 1670 a. As theuser 1670 a changes his pose, other visual graphics may become visible.For example, when the user 1670 a turns around, the user can perceivethe visual graphic 1610 a but the visual graphics 1630 a and 1640 a mayno longer be perceivable because they are outside of the user's FOV.

As described with reference to FIG. 13 , as the user 1670 a moves aroundin his environment, the appearance of the visual graphics may changeaccordingly. For example, as the user 1670 a approaches the visualgraphic 1610 a, the wearable system may present more details of thevisual graphic 1610 a. The wearable system can also emphasize the visualgraphic 1610 a, for example, to make it appear bigger and brighter, whenthe user is looking at the visual graphic 1610 a. However, as the userturns to a different direction, the visual graphic 1610 a may reduce thesize or brightness, while another visual graphic (such as 1650 b) in theuser's direction of gaze may be emphasized (as shown in FIG. 16B).

As another example, the user 1670 a may initially perceive visualgraphics 1630 a and 1640 a in his FOV. But as the user 1670 a movescloser to the visual graphic 1630 a, the visual graphic 1640 a may beoutside of the user's FOV and therefore the user 1670 a may no longer beable to perceive the visual graphic 1640.

In certain situations, the audio content may remain the same (e.g.,because the user has locked the spatial locations of the stem tracks)although the visual graphic may change based on the user's location. Forexample, the wearable system may make the visual graphic brighter whenit's closer to the user 1670 a while weaker when it's away from the user1670 a. In the meantime, the wearable system may be configured not toupdate the stem track associated with the visual graphic (such as notchanging the volume of the stem tack) as the user 1670 a moves around.

In other implementations, as the user 1670 a moves around in hisenvironment, the audio content in the audio file may be updated. Forexample, as the user 1670 a moves closer to the location of the visualgraphic 1610 a, the sound associated with the visual graphic 1610 a maybecome more prominent. Similarly, the user 1670 may initially face thevisual graphics 1630 a and 1640 a. As a result, the stem tracksassociated with the visual graphics 1630 a and 1640 a may be played asif they are in front of the user 1670 a. However, as the user 1670 aturns around, the stem tracks associated with the visual graphics 1630 aand 1640 a may be updated as if they are played behind the user 1670 a.

Directionalities of Sound Sources

FIGS. 16A and 16B also illustrate examples of directionalities of soundsources. As shown in the bird eye views 1600 a and 1600 b, the stemtracks 1610 a, 1620 a, and 1650 b may only be audible in a fewdirections. For example, the stem track 1610 a is only audible in theregion 1612 a; the stem track 1620 a is only audible in regions 1622 aand 1624 a; and the stem track 1650 b (shown in FIG. 16B) is onlyaudible in regions 1652 b, 1654 b, 1656 b, and 1658 b.

When a user is inside of the audible regions, the user can perceive thesounds associated with the audible regions. For example, in FIG. 16B,when the user is inside of the region 1652 b, the user can perceive thesound associated with the visual graphic 1650 b. However, if the user isoutside of the regions 1652 b, 1654 b, 1656 b, and 1658 b, the user maynot be able to hear the sound associated with the visual graphic 1650 b.

In some implementations, the audible regions for the stem tracks mayoverlap. As shown in FIG. 16A, the regions 1624 a and 1612 a overlap atregion 1672 a. As a result, when the user 1670 a is within the region1672 a, the wearable system may present a mixture of the soundsassociated with the visual graphics 1620 a and 1610 a. For example, theaudio mix may reflect the spatial location of the sounds. In the view1600 a, because the user is facing the visual graphic 1610 a, the soundassociated with the visual graphic 1610 a may appear to be in front ofthe user while the sound associated with the visual graphic 1622 a mayappear to be to the left of the user in the audio mix.

The wearable system may display the visual graphic 1610 a alone or incombination with visual graphic 1620 a in the FOV. For example, althoughthe user may hear the audio mix for sounds associated with visualgraphic 1610 a and 1620 a when the user is in the region 1672 a, theuser may only perceive the visual graphic 1610 a. In otherimplementations, wearable system may present both visual graphics (1610a and 1620 a) since the audio mix includes the sounds associated withboth visual graphics.

In some embodiments, the wearable system may be configured to onlydisplay visual graphics associated with the stem tracks when the user iswithin the audible regions of the stem tracks. As an example, thewearable system may display the visual graphic 1650 b when the user iswithin the regions 1652 b, 1654 b, 1656 b, and 1658 b. As the userapproaches the visual graphic 1650 b, the wearable system may show thevisual graphic 1650 b to be a larger and brighter. In the meantime, thewearable system may optionally increase the volume of the visual graphic1650 b in the audio mix. However, when the user is outside of audibleregions, the wearable system may be configured not shown the visualgraphic 1650 b, even though in some situations, the user is looking atthe direction of visual graphic 1650 b.

In certain implementations, only a subset of the stem tracks in theuser's environment may have the directional sound effect. For example,in FIG. 16A, the sounds associated with visual graphics 1630 a and 1640a may not have directional sound effects. As a result, while the user isin the region 1672 a, the wearable system may nevertheless include thesounds associated with visual graphics 1630 a and 1640 a, together withthe sounds associated with visual graphics 1620 a and 1610 a in theaudio mix.

The shapes and size of the sound regions in FIGS. 16A and 16B are forillustration purposes only and are not intended to be limiting. Theregion impacted by the directional sound effect may be associated withshapes other than lobes. The region may also be in bigger or smallerthan illustrated. For example, in some embodiments, there is no overlapamong the multiple sounds. As a result, a user can only perceive onesound source when he is within the region of that sound source.

Examples of Sound Collage Effects

FIG. 17 illustrates an example of creating a sound collage effect. FIG.17A illustrates a virtual 3D object 1700 which has multiple surfaces1710, 1720, and 1730. Each surface may be associated with an audio stem,an audio file, a type of sound, or other sound sources, alone or incombination. The audio stem on each surface may be directional, asdescribed above. For example, if a user is in front of the surface 1730,the user can hear the stem associated with the surface 1730, but theuser may not be able to hear the stem associated with the surface 1710because the sound associated the surface 1710 is not in the direction ofthe user.

The user of the wearable system can spin the 3D virtual object 1700 (ormove around it), such as in the direction 1750, to create an audio mixwhich includes sounds (or a portion thereof) associated with the surface1720, the surface 1730, and so on. In some implementations, the audiomix can correspond to movement of the 3D virtual object 1700. Forexample, where the user has stopped spinning the 3D virtual object 1700,the movement of the 3D virtual object 1700 may gradually slow down. As aresult, the sound collage may include a larger portion of the soundsassociated with certain surfaces.

Although the 3D virtual object 1700 is illustrated using a cubicalshape, other geometric shapes may also be used for the 3D virtualobject. For example, 3D virtual object may be a sphere, a cone, cuboid,polyhedron, pyramid, frustum, irregular shape, etc., alone or incombination. The stem tracks and/or sound files are also not required tobe associated with the surface of the virtual object. For example, theymay be associated with different volume portions of the 3D virtualobject. In some implementations, there may be multiple stem tracksand/or sound files associated with a given surface and/or volume.

Example Processes of Presenting Stem Tracks

FIG. 18 illustrates an example process of presenting an audio filevisually and audibly. The process 1800 can be performed by the wearablesystem described herein.

At block 1810, the wearable system can access stem tracks associatedwith an audio file. The wearable system can identify the stem tracks inthe audio file. The wearable system can also access stem tracks stored,for example, at the remote data repository 280 (shown in FIG. 2 ), whichmay not be part of the audio file but my be used to mix with other stemtracks of the audio file.

At block 1820, the wearable system generates a visualization of the stemtracks. For example, the wearable system can provide visual graphicsassociated with the stem tracks. As described with reference to FIG.12-14 , the visual graphic may represent the musical instrument or theperson playing the stem track or be any type of graphic shape, design,color, etc. A graphic may appear 2D or 3D. The visual graphics maychange based on the audio content of the music. For example, the visualgraphics may expand or shrink based on the volume, beats of the music,and so on.

The wearable system can also allow user interactions with the stemtracks using poses and/or user input device. At block 1830, the wearablesystem can detect a pose of the user. The pose may include handgestures, head pose, foot pose, or other body poses. The wearable systemcan detect the pose of the user using the sensors (such as the IMUs andthe outward-facing imaging system) described with reference to FIG. 2 .

At block 1840, the wearable system can generate an audio mix of the stemtracks based on the pose of the user. For example, a user may grab thevisual graphic of a stem track and cause a collision between two visualgraphics. The wearable system can accordingly mix their associated stemtracks to produce the audio mix. As another example, the user can turnhis body and perceive one or more visual graphics in his FOV. Thewearable system can mix the stem tracks associated with the visualgraphics in his FOV to generate the audio mix. As yet another example,the user may adjust a stem track, such as by adjusting the volumes orthe spatial location of the stem track using poses. The wearable systemcan accordingly produce the audio mix based on the adjustment to thestem track.

At block 1850, the wearable system can update the visualization based onthe pose of the user. For example, the wearable system can display asubset of the visual graphics in the user's FOV based on the user'sposition and orientation. The wearable system can also highlight oremphasize (or de-emphasize) one or more visual graphics based on theuser's pose. When the two or more visual graphics collide, the wearablesystem can also generate and display a new visual graphic. The newvisual graphic may reflect the collision between previous visualgraphics, for example, by interleaving the collided visual graphicstogether.

The wearable system can also allow multiple users to share theinteractions with the visual graphics and stem tracks. For example,there may be multiple users in a room. Each user can mix the stem tracksand the resulting audio and visual mix may be presented to all the usersin the room. In some implementations, the wearable system for each usercan present individualized visual graphics based on the user's positionand/or orientation. For example, a user facing east may perceive adifferent subset of visual graphics than a user facing west. As anotherexample, a user (such as a DJ) can interact with stem tracks in hisenvironment to create an audio mix. The wearable system of the user cancommunicate the audio mix to other wearable systems for streaming orplayback. The wearable system can also broadcast the audio mix to viaother computing systems. For example, the wearable system can play theaudio mix to other people via speakers in the room.

Although the examples described herein are described with reference tostem tracks, the techniques described herein are not limited to stemtracks. The wearable system can also present and allow user interactionswith other audio content. For example, the wearable system can allow auser to mix two audio files, where one audio file has background musicwhile the other audio file includes vocal.

Additional Embodiments

In a 1st aspect, a wearable display device comprising non-transitorymemory configured to measure information associated with a pose of thewearer of the wearable display device; a display configured to presentimages to an eye of the wearer of the wearable display device; a speakerconfigured to present sounds to the wearer of the wearable displaydevice; and a processor in communication with the non-transitory memory,the sensor, the speaker, and the display, the processor programmed withexecutable instructions to: determine the pose of the wearer; generate,based at least partly on the pose of the wearer, an audio mix of atleast one of the plurality of stem tracks of the audio file; generate,based at least partly on the pose of the wearer and the audio mix, avisualization of the audio mix; communicate an audio signalrepresentative of the audio mix to the speaker; and communicate a visualsignal representative of the visualization of the audio mix to thedisplay.

In a 2nd aspect, the wearable display device of aspect 1, wherein theplurality of stem tracks comprises at least four stem tracks.

In a 3rd aspect, the wearable display device of aspect 1 or aspect 2,wherein the plurality of stem tracks represent a plurality of tracks fordifferent musical instruments.

In a 4th aspect, the wearable display device of aspect 3, wherein theplurality of stem tracks further represents a vocal track.

In a 5th aspect, the wearable display device of aspect 4, wherein theplurality of stem tracks represents a guitar, a bass guitar, a drum, anda voice.

In a 6th aspect, the wearable display device of any one of aspects 1-5,wherein the sensor comprises an eye-tracking camera, a gyroscope, anaccelerometer, or an inertial measurement device.

In a 7th aspect, the wearable display device of any one of aspects 1-6,wherein the pose comprises a head pose, an eye pose, or both a head poseand an eye pose.

In an 8th aspect, the wearable display device of any one of aspects 1-7,wherein the pose comprises a gesture of the wearer.

In a 9th aspect, the wearable display device of any one of aspects 1-8,wherein the display is configured to present a plurality of depth planesto the wearer.

In a 10th aspect, the wearable display device of any one of aspects 1-9,wherein the display is configured to present a light field image to thewearer.

In an 11th aspect, the wearable display device of any one of aspects1-10, wherein the display comprises a plurality of stacked waveguides.

In a 12th aspect, the wearable display device of any one of aspects1-11, wherein the visualization comprises a visual graphic associatedwith each of the plurality of stem tracks.

In a 13th aspect, the wearable display device of aspect 12, wherein thevisual graphic comprises a graphical representation of an instrument orperson associated with performance of the stem track.

In a 14th aspect, the wearable display device of aspect 12 or aspect 13,wherein at least some of the visual graphics are presented at differentdepth planes relative to the wearer.

In a 15th aspect, the wearable display device of any one of aspects12-14, wherein the wearable display device is configured to recognize aphysical object in the vicinity of the wearer, and to present agraphical representation as being associated with the physical object.

In a 16th aspect, the wearable display device of any one of aspects1-15, wherein the wearable display device is configured to receive inputfrom an electronic component, and the processor is configured togenerate the audio mix or the visualization based at least in part onthe input.

In a 17th aspect, the wearable display device of any one of aspects1-16, wherein at least one of the plurality of stem tracks comprisesinformation associated with a spatial directionality of the stem track.

In a 18th aspect, the wearable display device of any one of aspects1-17, wherein at least one of the plurality of stem tracks is associatedwith a spatial location relative to the wearer.

In a 19th aspect, the wearable display device of aspect 18, wherein theprocessor is configured to lock the spatial location associated with theat least one of the plurality of stem tracks based at least partly onthe pose of the wearer or an input from an electronic component.

In a 20th aspect, the wearable display device of any one of aspects1-19, wherein the audio file is encoded in an audio format comprisingmetadata indicating at least one of: the plurality of stem tracks,positional data associated with the plurality of stem tracks at a giventime, or information on the visualization of the audio mix. In anotheraspect, an audio file is encoded in an audio format comprising metadataindicating at least one of: the plurality of stem tracks, positionaldata associated with the plurality of stem tracks at a given time, orinformation on the visualization of the audio mix. The audio file cancomprise information and/or identifiers associated with the plurality ofstem tracks. The audio file can include an audio mix of the plurality ofstem tracks.

In a 21st aspect, the wearable display device of any one of aspects1-20, wherein the visualization of the audio mix comprises athree-dimensional virtual object comprising a plurality of surfaceswherein each surface is associated with a stem track of the plurality ofstem tracks. The audio mix can comprise a sound collage effect of theplurality of stem tracks.

In a 22nd aspect, a method for interacting with an augmented realityobject, the method comprising: under control of a hardware computerprocessor: generating an augmented reality object for interaction by awearer of the wearable display system, wherein the augmented realityobject includes an interface for modification of a characteristicassociated with the augmented reality object and displayed on a displayof the wearable display system; detecting gestures of a wearer while thewearer interacts with the interface; associating the detected gestureswith a modification to a characteristic of the augmented reality object;and modifying the augmented reality object in accordance with themodification to the characteristic of the augmented reality object.

In a 23rd aspect, the method of aspect 22, wherein detecting thegestures of the wearer while the wearer interacts with the interfacecomprises: capturing, with an outward facing image capture device of thewearable display system, a first image of a hand of the wearer;capturing, with the outward facing image capture device of the wearabledisplay system, a second image of a hand of the wearer; and comparingthe first image and the second image to detect a gesture of the wearer.

In a 24th aspect, the method of any one of aspects 22-23, wherein theaugmented reality object represents a telephone, and wherein theinterface corresponds to the number keypad.

In a 25th aspect, the method of any one of aspects 22-24, whereinassociating the detected gestures with a modification to acharacteristic of the augmented reality object comprises associatingfinger depressions on the telephone to modify the appearance of thenumber keypad.

In a 26th aspect, the method of any one of aspects 22-25, whereinmodifying the augmented reality object in accordance with themodification to the characteristic of the augmented reality objectcomprises representing the number keypad with sequential depressionsdisplay on the keypad corresponding to the associated fingerdepressions.

In a 27th aspect, the method of any one of aspects 22-26, wherein theinterface of the augmented reality object corresponds to a skeuomorphicrepresentation of the augmented reality object.

In a 28th aspect, the method of any one of aspects 22-27, wherein theaugmented reality object corresponds to a musical instrument or a personplaying a musical instrument.

In a 29th aspect, the method of any one of aspects 22-28, whereinassociating the detected gestures with the modification to the musicalinstrument or the person playing the musical instrument comprisedetecting hand or finger movements of the wearer interacting with themusical instrument to increase an attribute of a stem track associatedwith the musical instrument.

In a 30th aspect, the method of any one of aspect 22-29, wherein theaugmented reality object is associated with an audio file comprising atleast one of a plurality of stem tracks.

In a 31st aspect, the method of aspect 30, wherein the augmented realityobject is a three-dimensional virtual object comprising a plurality ofsurfaces wherein each surface is associated with a stem track of theplurality of stem tracks.

In a 32nd aspect, the method of aspect 31, further comprises creating asound collage effect using the plurality of stem tracks in response tothe gestures of the wearer.

In a 33rd aspect, the method of any one of aspect 30-32, wherein thestem track associated with the musical instrument is one of a pluralityof stem tracks being played by an audio system coupled to the wearabledisplay system.

In a 34th aspect, the method of any one of aspect 30-33, wherein atleast one of the plurality of stem tracks is an original stem trackreplaced by a user generated stem track.

In a 35th aspect, the method of any one of aspect 30-34, wherein anadditional augmented reality object is selected with a head gesture ofthe wearer for alteration of the corresponding stem track represented bythe additional augmented reality object.

In a 36th aspect, the method of any one of aspects 30-35, wherein atleast one of the plurality of stem tracks comprises informationassociated with directionality of the stem track.

In a 37th aspect, the method of any one of aspects 30-36, wherein atleast one of the plurality of stem tracks is associated with a spatiallocation relative to the wearer.

In a 38th aspect, the method of aspect 37, further comprises locking thespatial location associated with the at least one of the plurality ofstem tracks based at least partly on the gestures of the wearer.

In a 39th aspect, the method of any one of aspects 30-38, wherein theaudio file is encoded in an audio format comprising metadata indicatingat least one of: the plurality of stem tracks, positional dataassociated with the plurality of stem tracks at a given time, orinformation on visualizations of the plurality of stem tracks.

In a 40th aspect, a wearable display system for interacting with avisual graphic, the system comprising: a display configured to displaythe visual graphic; at least one outward facing camera configured toimage a region in front of the wearable display system; and a computerprocessor configured to communicate with a network and programmed to:generate the visual graphic for interaction by a wearer of a wearabledisplay system, wherein the visual graphic includes an interface formodification of a characteristic associated with the visual graphic;detect gestures of a wearer interacting with the wearable displaysystem; associate the detected gestures with a modification to acharacteristic of the visual graphic; and alter the visual graphic inaccordance with the modification to the characteristic of the visualgraphic.

In a 41st aspect, the wearable display system of aspect 40, wherein thevisual graphic is associated with at least one stem tracks.

In a 42nd aspect, the wearable display system of aspect 41, wherein thestem track is directional.

In a 43rd aspect, the wearable display system of any one of aspects41-42, wherein the stem track is associated with a spatial locationrelative to the wearer.

In a 44th aspect, the wearable display system of any one of aspects41-43, wherein the computer processor is further configured to lock thespatial location of the stem track based at least partly on the pose ofthe wearer or an input from an electronic component.

In a 45th aspect, the wearable display system of any one of aspects41-44, wherein the visual graphic comprises a plurality of surfaceswherein each surface is associated with a stem track of a plurality ofstem tracks, and wherein the computer processor is further configured tocreate a sound collage effect of the plurality of stem tracks inresponse to detected gestures of the wearer.

In a 46th aspect, the wearable display device of any one of aspects1-21, the method of any one of aspects 22-39, the wearable displaysystem of any one of aspects 40-45, wherein the wearable display deviceand/or the wearable display system comprises a head-mounted display. Inany of these aspects, the wearable display device and/or the wearabledisplay system may comprise a light field display.

In a 47th aspect, wearable device comprising: non-transitory memoryconfigured to store an audio file comprising a plurality of stem tracks,with each stem track representing different audio content of the audiofile; a sensor configured to measure information associated with a poseof a user of the wearable device; a display configured to present imagesto an eye of the user of the wearable device; a speaker configured topresent sounds to the user of the wearable device; and a processor incommunication with the non-transitory memory, the sensor, the speaker,and the display, the processor programmed with executable instructionsto: determine the pose of the user; generate, based at least partly onthe pose of the user, an audio mix of at least one of the plurality ofstem tracks of the audio file; generate, based at least partly on thepose of the user and the audio mix, a visualization of the audio mix;communicate an audio signal representative of the audio mix to thespeaker; and communicate a visual signal representative of thevisualization of the audio mix to the display.

In a 48th aspect, the wearable device of aspect 47, wherein stem tracksof the plurality of stem tracks are associated with different musicalinstruments or a vocal track.

In a 49th aspect, the wearable device of any one of aspects 47-48,wherein at least one stem track of the plurality of stem trackscomprises information associated with a spatial location and a spatialdirectionality of the stem track.

In a 50th aspect, the wearable device of any one of aspects 47-49,wherein the sensor comprises at least one of: gyroscope, anaccelerometer, an inertial measurement unit, an eye tracking camera, ora camera configured to image an environment of the user.

In a 51st aspect, the wearable device of any one of aspects 47-50,wherein the pose comprises at least one of: a head pose, an eye pose, abody pose, or a hand gesture.

In a 52nd aspect, the wearable device of any one of aspects 47-51,wherein the visualization comprises visual graphics associated withrespective stem tracks of the plurality of stem tracks.

In a 53rd aspect, the wearable device of aspect 52, wherein the visualgraphic comprises a graphical representation of an instrument or aperson associated with performance of the stem track.

In a 54th aspect, the wearable device of aspect 52, wherein at leastsome of the visual graphics are presented at different depth planesrelative to the user.

In a 55th aspect, the wearable device of any one of aspects 47-54,wherein the wearable device is configured to recognize a physical objectin an environment of the user, and to present a graphical representationof a stem track of the plurality of stem tracks as being associated withthe physical object.

In a 56th aspect, the wearable device of any one of aspects 47-55,wherein the wearable device is configured to receive input from a userinput device, and the processor is configured to generate the audio mixor the visualization based at least in part on the input.

In a 57th aspect, a method for interacting with an audio file, themethod comprising: under control of a wearable device comprisinghardware computer processor, a sensor configured to measure informationassociated with a pose of a user of the wearable device; a displayconfigured to present images to an eye of the user of the wearabledevice; a speaker configured to present sounds to the user of thewearable device: accessing a plurality of stem tracks associated with anaudio file; generating a visualization of the plurality of stem tracks;detecting the pose of the user; generating, based at least partly on thepose of the user, an audio mix of at least one of the plurality of stemtracks of the audio file; updating the visualization of the plurality ofstem tracks based at least on one of the pose of the user or the audiomix; communicating an audio signal representative of the audio mix tothe speaker; and communicating a visual signal representative of thevisualization of the audio mix to the display.

In a 58th aspect, the method of aspect 57, wherein stem tracks of theplurality of stem tracks are associated with different musicalinstruments or a vocal track.

In a 59th aspect, the method of any one of aspects 57-58, wherein atleast one stem track of the plurality of stem tracks comprisesinformation associated with a spatial location and a spatialdirectionality of the stem track.

In a 60th aspect, the method of any one of aspects 57-59, wherein thevisualization comprises visual graphics corresponding to respective stemtracks of the plurality of stem tracks.

In a 61st aspect, the method of aspect 60, wherein the visual graphicscomprising musical instruments associated with the respective stemtracks or persons playing the musical instruments associated with therespective stem tracks.

In a 62nd aspect, the method of aspect 60, wherein at least some of thevisual graphics are presented at different depth planes relative to theuser.

In a 63rd aspect, the method of any one of aspects 57-62, wherein thepose comprises at least one of: a head pose, an eye pose, a body pose,or a hand gesture.

In a 64th aspect, the method of any one of aspects 57-63, whereinupdating the visualization comprises emphasizing or de-emphasizing avisual graphic associated with a stem track of a plurality of stemtracks.

In a 65th aspect, the method of any one of aspects 57-64, furthercomprising: receiving a selection of a stem track of the plurality ofstem tracks; receiving an indication to modify the stem track, whereinthe indication comprises a change in the pose of the user or an inputfrom a user input device; and modifying the stem track based at leastpartly on the indication.

In a 66th aspect, the method of aspect 65, where in modifying the stemtrack comprising at least one of: adjusting the volume, replacing thestem track with another stem track, muting the stem track, emphasizingthe stem track over other stem tracks of the plurality of stem tracks.

CONCLUSION

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A method for interacting with an augmentedreality object, the method comprising, under control of a hardwarecomputer processor: generating an augmented reality object forinteraction by a wearer of a wearable display system, wherein theaugmented reality object includes an interface for modification of acharacteristic associated with the augmented reality object anddisplayed on a display of the wearable display system and wherein theaugmented reality object is associated with an audio file comprising atleast one of a plurality of stem tracks; detecting gestures of thewearer while the wearer interacts with the interface; associating thedetected gestures with a modification to a characteristic of theaugmented reality object; modifying the augmented reality object inaccordance with the modification to the characteristic of the augmentedreality object; and playing audiovisual data comprising at least one ofthe plurality of stem tracks that is associated with the modifiedaugmented reality object via at least a speaker of the wearable displaysystem.
 2. The method of claim 1, wherein detecting the gestures of thewearer while the wearer interacts with the interface comprises:capturing, with an outward facing image capture device of the wearabledisplay system, a first image of a hand of the wearer; capturing, withthe outward facing image capture device of the wearable display system,a second image of a hand of the wearer; and comparing the first imageand the second image to detect a gesture of the wearer.
 3. The method ofclaim 1, wherein the interface of the augmented reality objectcorresponds to a skeuomorphic representation of the augmented realityobject.
 4. The method of claim 1, wherein the augmented reality objectcorresponds to a musical instrument or a person playing a musicalinstrument.
 5. The method of claim 1, wherein associating the detectedgestures with the modification to the musical instrument or the personplaying the musical instrument comprise detecting hand or fingermovements of the wearer interacting with the musical instrument tomodify an attribute of a stem track associated with the musicalinstrument.
 6. The method of claim 1, wherein associating the detectedgestures with the modification to the musical instrument or the personplaying the musical instrument comprise detecting hand or fingermovements of the wearer interacting with the musical instrument togenerate a new stem track associated with the musical instrument.
 7. Themethod of claim 1, wherein the augmented reality object is athree-dimensional virtual object comprising a plurality of surfaceswherein each surface is associated with a stem track of the plurality ofstem tracks.
 8. The method of claim 7, further comprises creating asound collage effect using the plurality of stem tracks in response tothe gestures of the wearer.
 9. The method of claim 5, wherein the stemtrack associated with the musical instrument is one of a plurality ofstem tracks being played by an audio system coupled to the wearabledisplay system.
 10. The method of claim 1, wherein at least one of theplurality of stem tracks is an original stem track replaced by a usergenerated stem track.
 11. The method of claim 1, wherein an additionalaugmented reality object is selected with a head gesture of the wearerfor alteration of the corresponding stem track represented by theadditional augmented reality object.
 12. The method of claim 1, whereinat least one of the plurality of stem tracks comprises informationassociated with directionality of the stem track.
 13. The method ofclaim 1, wherein at least one of the plurality of stem tracks isassociated with a spatial location relative to the wearer.
 14. Themethod of claim 16, further comprises locking the spatial locationassociated with the at least one of the plurality of stem tracks basedat least partly on the gestures of the wearer.
 15. The method of claim1, wherein the audio file is encoded in an audio format comprisingmetadata indicating at least one of: the plurality of stem tracks,positional data associated with the plurality of stem tracks at a giventime, or information on visualizations of the plurality of stem tracks.16. A wearable device comprising: non-transitory memory configured tostore an audio file comprising a plurality of stem tracks, with eachstem track representing different audio content of the audio file; asensor configured to measure information associated with a pose of auser of the wearable device; a display configured to present images toan eye of the user of the wearable device; a speaker configured topresent sounds to the user of the wearable device; and a processor incommunication with the non-transitory memory, the sensor, the speaker,and the display, the processor programmed with executable instructionsto: detect gestures of the user while the user interacts with theinterface; associate the detected gestures with a modification to acharacteristic of the augmented reality object; modify the augmentedreality object in accordance with the modification to the characteristicof the augmented reality object; and play audiovisual data comprising atleast one of the plurality of stem tracks that is associated with themodified augmented reality object via at least a speaker of the wearabledisplay system.
 17. The wearable device of claim 16, wherein to detectthe gestures of the wearer while the wearer interacts with theinterface, the wearable device is configured to: capture, with anoutward facing image capture device of the wearable display system, afirst image of a hand of the wearer; capture, with the outward facingimage capture device of the wearable display system, a second image of ahand of the wearer; and compare the first image and the second image todetect a gesture of the wearer.
 18. The wearable device of claim 16,wherein the augmented reality object corresponds to a musical instrumentor a person playing a musical instrument.
 19. The wearable device ofclaim 16, wherein to associate the detected gestures with themodification to the musical instrument or the person playing the musicalinstrument, the processor is configured to detect hand or fingermovements of the wearer interacting with the musical instrument tomodify an attribute of a stem track associated with the musicalinstrument.
 20. The wearable device of claim 16, wherein to associatethe detected gestures with the modification to the musical instrument orthe person playing the musical instrument, the processor is configuredto detect hand or finger movements of the wearer interacting with themusical instrument to generate a new stem track associated with themusical instrument.
 21. The method of claim 18, wherein the stem trackassociated with the musical instrument is one of a plurality of stemtracks being played by an audio system coupled to the wearable displaysystem.
 22. The wearable device of claim 16, wherein at least one of theplurality of stem tracks is an original stem track replaced by a usergenerated stem track.
 23. The wearable device of claim 16, wherein anadditional augmented reality object is selected with a head gesture ofthe wearer for alteration of the corresponding stem track represented bythe additional augmented reality object.
 24. The wearable device ofclaim 28, wherein the processor is configured to lock the spatiallocation associated with the at least one of the plurality of stemtracks based at least partly on the gestures of the wearer.
 25. Thewearable device of claim 16, wherein the audio file is encoded in anaudio format comprising metadata indicating at least one of: theplurality of stem tracks, positional data associated with the pluralityof stem tracks at a given time, or information on visualizations of theplurality of stem tracks.